fix(sonarqube): increase cq_issues and cq_file_metrics project_key length to 500 by jbsmith7741 · Pull Request #8783 · apache/devlake

jbsmith7741 · 2026-03-19T15:04:51Z

Summary

The project_key column in cq_issues and cq_file_metrics were too short to hold the domain-layer project ID generated by didgen (format: sonarqube:SonarqubeProject::). Since SonarQube allows project keys up to 400 characters, the generated domain ID can be around ~450 characters, which will cause the "Data too long for column" error. This changes the columns cq_issues and cq_file_metrics to varchar(500) which allows for the long names and is consistent with DomainEntityExtended.Id

Does this close any open issues?

Closes #8331

Screenshots

N/A

Other Information

Added migration script (20260317_increase_cq_issues_project_key_length) to alter both columns on the next startup.

…ngth to 500 Fixes apache#8331

klesh

LGTM

…ngth to 500 (apache#8783) Fixes apache#8331

…ency (#8811) * fix: Add backfill time window to ensure data consistency chore: Add backfill time window to ensure data consistency (cherry picked from commit 52a9555b782860f9c8ba9db409bd56e0c8f58272) * fix(q_dev): prevent data duplication in user_report and user_data tables (#8737) * fix(q_dev): prevent data duplication in user_report and user_data tables Replace auto-increment ID with composite primary keys so that CreateOrUpdate can properly deduplicate rows on re-extraction. - user_report PK: (connection_id, scope_id, user_id, date, client_type) - user_data PK: (connection_id, scope_id, user_id, date) - Switch db.Create() to db.CreateOrUpdate() in s3_data_extractor - Migration drops old tables, rebuilds with new PKs, resets s3_file_meta processed flag to trigger re-extraction * fix(q_dev): gofmt archived user_data_v2 model * feat(github): Extend exclusion of file extensions to github plugin (#8719) * feat(github): extend PR size exclusion for specified file extension to github plugin * fix: register migration script * fix: move PR size to 'Additional settings' and change so the comma doesn't get removed while typing * fix: linting * fix(doc): update expired Slack invite links in README (#8739) The Slack invite links in README.md were expired and returning "This link is no longer active." Updated both occurrences (badge and community section) to match the current link on the official DevLake website. Closes #8738 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> * docs: add gh-devlake CLI to Getting Started installation options (#8733) Adds gh-devlake as a third installation method alongside Docker Compose and Helm. gh-devlake is a GitHub CLI extension that automates DevLake deployment, configuration, and monitoring from the terminal. Closes #8732 * fix(gitlab): add missing repos scope in project_mapping (#8743) GitLab's makeScopeV200 did not create a repos scope when scopeConfig.Entities was empty or only contained CROSS. This caused project_mapping to have no table='repos' row, breaking downstream DORA metrics, PR-issue linking, and all PR dashboard panels that join on project_mapping. The fix aligns GitLab with the GitHub plugin by: 1. Defaulting empty entities to plugin.DOMAIN_TYPES 2. Adding DOMAIN_TYPE_CROSS to the repo scope condition Closes #8742 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> * fix(grafana): update dashboard descriptions to list all supported data sources (#8741) Several dashboard introduction panels hardcoded "GitHub and Jira" as required data sources, even though the underlying queries use generic domain layer tables that work with any supported Git tool or issue tracker. Updated to list all supported sources following the pattern already used by DORA and WorkLogs dashboards. Closes #8740 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> * fix: modify cicd_deployments name from varchar to text (#8724) * fix: modify cicd_deployments name from varchar to text * fix: update the year * fix(q_dev): replace MariaDB-specific IF NOT EXISTS syntax with DAL methods for MySQL 8.x compatibility (#8745) * fix(azuredevops): default empty entities and add CROSS to repo scope in makeScopeV200 (#8751) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopeV200 produced zero scopes, leaving project_mapping with no rows. Additionally, the repo scope condition did not check for DOMAIN_TYPE_CROSS, so selecting only CROSS would not create a repo scope, breaking DORA metrics. This adds the same fixes applied to GitLab in #8743. Closes #8749 * fix(bitbucket): default empty entities to all domain types in makeScopesV200 (#8750) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopesV200 produced zero scopes, leaving project_mapping with no repo rows. This adds the same empty-entities default applied to GitLab in #8743. Closes #8748 * feat(circleci): add server version requirement and endpoint help text (#8757) Update CircleCI connection form to indicate Server v4.x+ requirement and provide guidance for server endpoint configuration. Signed-off-by: Joshua Smith <jbsmith7741@gmail.com> * feat(asana): add Asana plugin for project and task collection (#8758) Add a new Asana plugin that integrates with Asana's REST API to collect projects, sections, tasks, subtasks, stories (comments), tags, and users, mapping them to DevLake's ticket/board domain model. Backend: - Plugin implementation with all required interfaces (PluginMeta, PluginTask, PluginModel, PluginMigration, PluginSource, PluginApi, DataSourcePluginBlueprintV200) - Collectors, extractors, and converters for projects, sections, tasks, subtasks, stories, tags, and users - Remote API scope picker (Workspaces -> Teams/Portfolios -> Projects) - Scope config with issue-type regex transformation rules - Migration scripts for schema evolution - E2E tests with CSV fixtures for project and task data flows Config UI: - Plugin registration with connection form (PAT auth, endpoint, proxy) - Scope config transformation form for issue-type mapping - Dashboard URL integration for onboarding flow Grafana: - Asana dashboard with task metrics and visualizations Made-with: Cursor * feat: GitHub App token refresh (#8746) * feat(github): auto-refresh GitHub App installation tokens Add transport-level token refresh for GitHub App (AppKey) connections. GitHub App installation tokens expire after ~1 hour; this adds proactive refresh (before expiry) and reactive refresh (on 401) using the existing TokenProvider/RefreshRoundTripper infrastructure. New files: - app_installation_refresh.go: refresh logic + DB persistence - refresh_api_client.go: minimal ApiClient for token refresh POST - cmd/test_refresh/main.go: manual test script for real GitHub Apps Modified: - connection.go: export GetInstallationAccessToken, parse ExpiresAt - token_provider.go: add refreshFn for pluggable refresh strategies - round_tripper.go: document dual Authorization header interaction - api_client.go: wire AppKey connections into refresh infrastructure - Tests updated for new constructors and AppKey refresh flow * feat(github): add diagnostic logging to GitHub App token refresh Add structured logging at key decision points for token refresh: - Token provider creation (connection ID, installation ID, expiry) - Round tripper installation (connection ID, auth method) - Proactive refresh trigger (near-expiry detection) - Refresh start/success/failure (old/new token prefixes, expiry times) - DB persistence success/failure - Reactive 401 refresh and skip-due-to-concurrent-refresh All logs route through the DevLake logger to pipeline log files. * fix(github): prevent deadlock and fix token persistence in App token refresh Deadlock fix: NewAppInstallationTokenProvider now captures client.Transport (the base transport) before wrapping with RefreshRoundTripper. The refresh function uses newRefreshApiClientWithTransport(baseTransport) to POST for new installation tokens, bypassing the RefreshRoundTripper entirely. Token persistence fix: PersistEncryptedTokenColumns() manually encrypts tokens via plugin.Encrypt() then writes ciphertext via dal.UpdateColumns with conn.TableName() (a string) as the first argument. Passing the table name string makes GORM use Table() instead of Model(), preventing the encdec serializer from corrupting the in-memory token value. The encryption secret is threaded from taskCtx.GetConfig(ENCRYPTION_SECRET) through CreateApiClient to TokenProvider to persist functions. Also persists the initial App token at startup for DB consistency, and adds TestProactiveRefreshNoDeadlock with a real RSA key to verify the deadlock scenario is resolved. * fix(grafana): update dashboard descriptions to list all supported data sources (#8741) Several dashboard introduction panels hardcoded "GitHub and Jira" as required data sources, even though the underlying queries use generic domain layer tables that work with any supported Git tool or issue tracker. Updated to list all supported sources following the pattern already used by DORA and WorkLogs dashboards. Closes #8740 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> * fix: modify cicd_deployments name from varchar to text (#8724) * fix: modify cicd_deployments name from varchar to text * fix: update the year * fix(q_dev): replace MariaDB-specific IF NOT EXISTS syntax with DAL methods for MySQL 8.x compatibility (#8745) * fix(azuredevops): default empty entities and add CROSS to repo scope in makeScopeV200 (#8751) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopeV200 produced zero scopes, leaving project_mapping with no rows. Additionally, the repo scope condition did not check for DOMAIN_TYPE_CROSS, so selecting only CROSS would not create a repo scope, breaking DORA metrics. This adds the same fixes applied to GitLab in #8743. Closes #8749 * fix(bitbucket): default empty entities to all domain types in makeScopesV200 (#8750) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopesV200 produced zero scopes, leaving project_mapping with no repo rows. This adds the same empty-entities default applied to GitLab in #8743. Closes #8748 * fix(github): remove unused refresh client constructor and update tests --------- Co-authored-by: Spiff Azeta <35563797+spiffaz@users.noreply.github.com> Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> Co-authored-by: Dan Crews <crewsd@gmail.com> Co-authored-by: Tomoya Kawaguchi <68677002+yamoyamoto@users.noreply.github.com> * fix: cwe89 sql injection (#8762) * feat(q-dev): add logging data ingestion and enrich Kiro dashboards (#8767) * feat(q-dev): add logging data ingestion and enrich Kiro dashboards Add support for ingesting S3 logging data (GenerateAssistantResponse and GenerateCompletions events) into new database tables, and enrich all three Kiro Grafana dashboards with additional metrics. Changes: - New models: QDevChatLog and QDevCompletionLog for logging event data - New extractor: s3_logging_extractor.go parses JSON.gz logging files - Updated S3 collector to also handle .json.gz files - Added logging S3 prefixes (GenerateAssistantResponse, GenerateCompletions) - New dashboard: "Kiro AI Activity Insights" with 10 panels including model usage distribution, active hours, conversation depth, feature adoption (Steering/Spec), file type usage, and prompt/response trends - Enriched "Kiro Code Metrics Dashboard" with DocGeneration, TestGeneration, and Dev (Agentic) metric panels - Fixed "Kiro Usage Dashboard" per-user table to sort by user_id - Migration script for new tables * fix(q-dev): use separate base path for logging S3 prefixes Logging data lives under a different S3 prefix ("logging/") than user report data ("user-report/"). Add LoggingBasePath option (defaults to "logging") so logging prefixes are constructed correctly. * fix(q-dev): auto-scan logging path without extra config Kiro exports to two well-known S3 prefixes in the same bucket: - user-report/AWSLogs/{accountId}/KiroLogs/ (CSV reports) - logging/AWSLogs/{accountId}/KiroLogs/ (interaction logs) When AccountId is set, automatically scan both paths. The "logging" prefix is hardcoded since it's a standard Kiro export convention. No additional configuration needed. * fix(q-dev): update scope tooltip to mention logging data scanning * fix(q-dev): fix scope ID routing and CSV/JSON file separation Three fixes: 1. Use *scopeId (catch-all) route pattern instead of :scopeId so scope IDs containing "/" (e.g. "034362076319/2026") work in URL paths 2. CSV extractor now filters for .csv files only, preventing it from trying to parse .json.gz logging files as CSV 3. Frontend scope API calls now encodeURIComponent(scopeId) for safe URL encoding * fix(q-dev): resolve *scopeId route conflict with dispatcher pattern The catch-all *scopeId route conflicts with *scopeId/latest-sync-state. Follow Jenkins/Bitbucket pattern: use a single *scopeId route with a GetScopeDispatcher that checks for /latest-sync-state suffix and dispatches accordingly. All scope handlers now TrimLeft "/" from scopeId. * fix(q-dev): use URL-safe scope ID format (underscore separator) Scope IDs like "034362076319/2026" break URL routing because "/" is a path separator. Change ID format to "034362076319_2026" (underscore) when AccountId is set. The Prefix field still uses "/" for S3 path matching. Revert to standard :scopeId routes since IDs are now safe. Note: existing scopes need to be recreated after this change. * fix(q-dev): use NoPKModel instead of Model in archived logging models archived.Model only has ID+timestamps, missing RawDataOrigin fields (_raw_data_params etc.) that common.NoPKModel includes. This caused "Unknown column '_raw_data_params'" errors at runtime. * fix(q-dev): fix GROUP BY in per-user table to merge display_name variants Remove display_name from GROUP BY so same user_id with different display_name values gets merged. Use MAX(display_name) in SELECT. * fix(q-dev): normalize logging user IDs to match CSV short UUID format Logging data uses "d-{directoryId}.{UUID}" format while CSV user-report uses plain "{UUID}". Strip the "d-xxx." prefix so the same user maps to one user_id across both data sources. * fix(q-dev): normalize user IDs in CSV extractors and sort table DESC Apply normalizeUserId to both createUserReportData and createUserDataWithDisplayName so user_report CSV data also strips the "d-{directoryId}." prefix. Change per-user table sort to ORDER BY user_id DESC. * style(q-dev): fix gofmt formatting in chat_log models * perf(q-dev): parallelize logging S3 downloads and batch DB writes Optimize logging extractor performance: - 10 goroutine workers for parallel S3 file downloads - Batch 50 files per DB transaction instead of 1-per-file - sync.Map cache for display name resolution (avoid repeated IAM calls) - Parse records in memory during download, write all at once This should improve throughput from ~1.5 files/sec to ~15+ files/sec for typical logging file sizes. * fix(q-dev): check tx.Rollback error return to satisfy errcheck lint * feat(q-dev): add per-user model usage table and models column Add "Per-User Model Usage" table (panel 11) showing each user's request count and avg prompt/response length per model_id. Also add "Models Used" column to the Per-User Activity table. * fix(q-dev): remove per-user model usage table, keep models column only * feat(q-dev): add Kiro Executive Dashboard with cross-source analytics New dashboard "Kiro Executive Dashboard" with 12 panels covering: - KPIs: WAU, credits efficiency, acceptance rate, steering adoption - Trends: weekly active users, new vs returning users - Adoption funnel: Chat→Inline→CodeFix→Review→DocGen→TestGen→Agentic→Steering→Spec - Cost: credits pace vs projected monthly, idle power users - Quality: acceptance rate trends, code review findings, test generation - Efficiency: per-user productivity table with credits/line ratio Correlates data across user_report (credits), user_data (code metrics), and chat_log (interaction patterns) for holistic Kiro usage insights. * fix(q-dev): fix pie charts to show per-row slices instead of single total Set reduceOptions.values=true so Grafana treats each SQL result row as a separate pie slice. Fixes Model Usage Distribution, File Type Usage, Kiro Feature Adoption, and Active File Types pie charts. * fix(q-dev): cast Hour to string for Active Hours bar chart x-axis * fix(q-dev): fix pie chart single-slice and GROUP BY display_name issues 1. qdev_user_report Panel 4 (Subscription Tier Distribution): set reduceOptions.values=true to show per-tier slices 2. qdev_user_data Panel 6 (User Interactions): remove display_name from GROUP BY, use MAX(display_name) to merge same user * fix(q-dev): prevent data inflation in user_report JOIN user_data user_report has multiple rows per (user_id, date) due to client_type (KIRO_IDE, KIRO_CLI), but user_data has only one row per (user_id, date). A direct JOIN causes user_data metrics to be counted multiple times. Fix: pre-aggregate user_report by (user_id, date) in a subquery before joining, so the JOIN is always 1:1. Affects: Credits Efficiency stat and User Productivity table. * feat(qa): add is_invalid field to qa_test_case_executions (#8764) * feat(qa): add is_invalid field to qa_test_case_executions Add is_invalid boolean field to the domain layer qa_test_case_executions table to allow QA teams to flag test executions as invalid due to environmental issues, flaky tests, false positives, or false negatives. Changes: - Add IsInvalid field to QaTestCaseExecution domain model - Create migration script (20260313_add_is_invalid_to_qa_test_case_executions) - Register migration in migrationscripts/register.go - Update customize service to set default value for is_invalid - Update E2E test data to include new column Resolves #8763 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(qa): handle missing is_invalid column in CSV import Fix PostgreSQL compatibility issue when CSV files don't contain the is_invalid column. The field now defaults to false instead of an empty string. Changes: - Update qaTestCaseExecutionHandler to check for empty string values - Add E2E test for backward compatibility with CSV files lacking is_invalid - Add explicit IsInvalid initialization in Testmo plugin converter Resolves #8763 --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat(linker): link when branch names contain issue keys (#8777) * feat(linker): branch names containing issue keys * chore: add testing data * Add codespell support with configuration and fixes (#8761) * ci(codespell): add codespell config and GitHub Actions workflow Add .codespellrc with skip patterns for generated files, camelCase/PascalCase ignore-regex, and project-specific word list (convertor, crypted, te, thur). Add GitHub Actions workflow to run codespell on push to main and PRs. Co-Authored-By: Claude Code 2.1.63 / Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Yaroslav Halchenko <debian@onerussian.com> * fix(codespell): fix ambiguous typos requiring context review Manual fixes for typos that needed human review to avoid breaking code: - Comment/string typos: occured->occurred, destory->destroy, writting->writing, retreive->retrieve, identifer->identifier, etc. - Struct field comments and documentation corrections - Migration script comment fixes (preserving Go identifiers like DataConvertor) Co-Authored-By: Claude Code 2.1.63 / Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Yaroslav Halchenko <debian@onerussian.com> * fix(codespell): fix non-ambiguous typos with codespell -w Automated fix via `codespell -w` for clear-cut typos across backend, config-ui, and grafana dashboards. Examples: sucess->success, occurence->occurrence, exeucte->execute, asynchornous->asynchronous, Grafana panel typos, etc. Co-Authored-By: Claude Code 2.1.63 / Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Yaroslav Halchenko <debian@onerussian.com> --------- Signed-off-by: Yaroslav Halchenko <debian@onerussian.com> Co-authored-by: Claude Code 2.1.63 / Claude Opus 4.6 <noreply@anthropic.com> * feat(q-dev): enrich logging fields, separate dashboards, add E2E tests (#8786) * feat(q-dev): enrich logging fields, separate dashboards by data source, add E2E tests - Add new fields to chat_log: CodeReferenceCount, WebLinkCount, HasFollowupPrompts (from codeReferenceEvents, supplementaryWebLinksEvent, followupPrompts in JSON) - Add new fields to completion_log: LeftContextLength, RightContextLength (from leftContext/rightContext in JSON) - Update s3_logging_extractor to parse and populate new fields - Add migration script 20260319_add_logging_fields - Create qdev_feature_metrics dashboard for legacy by_user_analytic data - Reorganize qdev_executive dashboard with Row dividers labeling data sources and cross-dashboard navigation links - Enrich qdev_logging dashboard with new panels: Chat Trigger Type Distribution, Response Enrichment Breakdown, Completion Context Size Trends, Response Enrichment Trends - Fix SQL compatibility with only_full_group_by mode in executive dashboard (Weekly Active Users Trend, New vs Returning Users) - Fix Steering Adoption stat panel returning string instead of numeric value - Add Playwright E2E test covering full pipeline flow and dashboard verification * fix: add Apache license headers to e2e files, fix gofmt alignment * fix: add SQL identifier validation to prevent SQL injection via table/column names (#8769) Add ValidateTableName and ValidateColumnName functions in core/dal to ensure table and column names used in dynamic SQL are safe identifiers. Applied to scope_service_helper, scope_generic_helper, and customized_fields_extractor. * feat(q-dev): add Kiro Credits + DORA Correlation dashboard (#8792) Add a new Grafana dashboard that correlates Kiro AI usage (credits, messages, active users) with DORA metrics at weekly aggregate level. Panels include: - Pearson's r correlation between weekly credits and PR cycle time - High AI Usage vs Low AI Usage cycle time comparison - Weekly credits vs deployment frequency trend - Weekly credits vs change failure rate trend Data is joined by week_start between _tool_q_dev_user_report and project_pr_metrics / cicd_deployment_commits. * feat(q-dev): add AI Cost-Efficiency dashboard (#8793) Add a Grafana dashboard showing AI tool cost-efficiency metrics: - Credits per merged PR (overall + weekly trend) - Credits per production deployment (overall + weekly trend) - Credits per issue resolved (overall + weekly trend) - Weekly AI activity volume (credits, messages, conversations) Joins _tool_q_dev_user_report with pull_requests, cicd_deployment_commits, and issues by weekly aggregation. * feat(q-dev): add Multi-AI Tool Comparison dashboard (Copilot vs Kiro) (#8794) Add a Grafana dashboard comparing GitHub Copilot and Kiro side by side: - Weekly active users comparison - Code suggestions & acceptance events (per tool) - LOC accepted comparison (combined time series) - Acceptance rate comparison (bar gauge) Template variables for Copilot connection/scope selection. Data from _tool_copilot_enterprise_daily_metrics vs _tool_q_dev_user_report and _tool_q_dev_user_data. * feat(q-dev): add Kiro AI Model ROI dashboard (#8795) Add a Grafana dashboard analyzing per-model performance from chat logs: - Model Performance Summary table (requests, share%, avg prompt/response length, response/prompt ratio, steering/spec mode usage) - Daily Model Usage Distribution (stacked bar chart) - Avg Response Length by Model trend (output quality proxy) Data source: _tool_q_dev_chat_log grouped by model_id. * feat(q-dev): add Steering & Spec Mode Adoption dashboard (#8798) Track Kiro steering rules and spec mode adoption: - User/request adoption rate stats - Weekly adoption rate trend - Steering impact on prompt/response length - Per-user feature adoption table * feat(q-dev): add Developer AI Productivity Hours dashboard (#8797) Analyze when developers are most productive with AI tools: - AI Activity by Hour of Day (chat + completions stacked bar) - Prompt & Response Length by Hour (complexity patterns) - Feature Usage by Hour (steering/spec mode/plain chat) - AI Activity by Day of Week * feat(q-dev): add Language AI Heatmap dashboard (#8796) Analyze AI-assisted coding patterns by programming language: - Language Completion Profile table (requests, avg completions, context sizes, users per language) - Daily Completions by Language (stacked bar) - Active File Types During Chat (donut) - Avg Context Size by Language trend (top 5) * Fix/circleci column names (#8799) * fix(circleci): rename created_at to created_date in jobs/workflows Add migration to copy created_at -> created_date and update models/converters. * fix(circleci): update pipeline parsing * test(circleci): add incremental tests for collectors * fix(jenkins): scope multi-branch build collection to current project (#8430) (#8781) The branch jobs query in collectMultiBranchJobApiBuilds selected all WorkflowJob entries across all multi-branch pipelines for a connection, causing builds to be duplicated and misattributed. Filter by _raw_data_params to collect only the current project's branch jobs. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: Make gh-copilot plugin database agnostic (#8779) Co-authored-by: Eldrick Wega <eldrick.wega@outlook.com> * fix(sonarqube): increase cq_issues and cq_file_metrics project_key length to 500 (#8783) Fixes #8331 * feat: added taiga plugin (#8755) * feat: added taiga plugin * fix: fixed tests * feat(gh-copilot): add support for organization daily user metrics (#8747) * feat(circleci): add server version requirement and endpoint help text (#8757) Update CircleCI connection form to indicate Server v4.x+ requirement and provide guidance for server endpoint configuration. Signed-off-by: Joshua Smith <jbsmith7741@gmail.com> * fix: fixed test files --------- Signed-off-by: Joshua Smith <jbsmith7741@gmail.com> Co-authored-by: Reece Ward <47779818+ReeceXW@users.noreply.github.com> Co-authored-by: Joshua Smith <jbsmith7741@gmail.com> * fix(docker): pin Poetry to 2.2.1 for Python 3.9 compatibility (#8735) Poetry 2.3.0 dropped Python 3.9 support. Without cache the installer fetches the latest version (currently 2.3.2), which fails on the python:3.9-slim-bookworm base image. Pin to 2.2.1, the last release compatible with Python 3.9. Co-authored-by: Rodrigo Silva <rodrigo.silva@bonial.com> * fix(linker): scope clearHistoryData to current project only (#8814) (#8815) The clearHistoryData() function used a LEFT JOIN with project_name in the ON clause, causing the subquery to return all PR IDs regardless of project. This effectively wiped the entire pull_request_issues table on every linker run, deleting links from other projects sharing the same repos and links created by the GitHub converter. Fix: - Use INNER JOIN + WHERE for proper project scoping - Add issue-side subquery scoped to current project's boards - Filter by _raw_data_table/_raw_data_remark to only delete linker-created rows Add e2e test for cross-project shared repo scenario. * fix(circleci): prevent negative values when calculating circleci (#8800) workflow duration * fix: sonarqube: missing api/users/search endpoint (#8813) * fix(argocd): extract revision from multi-source application revisions[] (#8810) --------- Signed-off-by: Joshua Smith <jbsmith7741@gmail.com> Signed-off-by: Yaroslav Halchenko <debian@onerussian.com> Co-authored-by: tamas.albert <tamas.laczkoalbert@concentrix.com> Co-authored-by: Warren Chen <warren.chen830@gmail.com> Co-authored-by: Ema Abitante <ema.abitante@gmail.com> Co-authored-by: Spiff Azeta <35563797+spiffaz@users.noreply.github.com> Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> Co-authored-by: Eldrick Wega <eldrick.wega@outlook.com> Co-authored-by: Dan Crews <crewsd@gmail.com> Co-authored-by: Tomoya Kawaguchi <68677002+yamoyamoto@users.noreply.github.com> Co-authored-by: Joshua Smith <jbsmith7741@gmail.com> Co-authored-by: jawad khan <jawadkhan444@gmail.com> Co-authored-by: Leif Roger Frøysaa <leif.roger.froysaa@akerbp.com> Co-authored-by: Klesh Wong <klesh@qq.com> Co-authored-by: NaRro <cong.wang@merico.dev> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Reece Ward <47779818+ReeceXW@users.noreply.github.com> Co-authored-by: Yaroslav Halchenko <debian@onerussian.com> Co-authored-by: Chris Pavlicek <varsis@users.noreply.github.com> Co-authored-by: AvivGuiser <avivguiser@gmail.com> Co-authored-by: Shayne Clausson <shayne.clausson@extendaretail.com> Co-authored-by: irfanuddinahmad <34648393+irfanuddinahmad@users.noreply.github.com> Co-authored-by: Rodrigo Silva <rodrigoluizscs@gmail.com> Co-authored-by: Rodrigo Silva <rodrigo.silva@bonial.com> Co-authored-by: Daniele M. <github@dmoraschi.com> Co-authored-by: Pavel Sturc <psturc@redhat.com> Co-authored-by: Anvesh Vemula <39478419+vemulaanvesh@users.noreply.github.com>

* chore(deps): update graphql dependency from merico-dev to merico-ai (#8699) * chore(deps): update graphql dependency from merico-dev to merico-ai - Update github.com/merico-dev/graphql to github.com/merico-ai/graphql - Update import paths across all files using the graphql package - Update go.mod and go.sum with new dependency version Co-Authored-By: Claude <noreply@anthropic.com> * fix(github): fix ineffassign lint error in remote_api.go Use named return values consistently with bare returns to avoid variable shadowing of err in listGithubOwnerRepos. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> * fix(github): made token expiration fields nullable and updated related logic (#8707) * fix(plugins): add missed DomainTypes in ArgoCD plugin (#8706) * fix(plugins): add missed repoURL in ArgoCD plugin (#8712) * feat(q_dev): add Account ID to scope config and auto-construct both report paths (#8725) User now provides basePath and accountId instead of manually typing the full S3 prefix. The collector automatically constructs and scans both by_user_analytic and user_report paths under {basePath}/AWSLogs/{accountId}/KiroLogs/…/{region}/{year}[/{month}]. Includes migration, updated blueprint, multi-prefix collector iteration, user_report model/extractor/dashboard, and frontend Account ID input field. Old scopes without accountId continue to work unchanged. * Fix/GitHub deployment time calculation (#8715) * feat(github_graphql): add finished_date to GithubDeployment model Introduce FinishedDate field in GithubDeployment to represent success-priority deployment completion timestamp. This field allows distinguishing SUCCESS timestamp from latest status update time while preserving backward compatibility. * feat(github_graphql): compute success-priority deployment finish time Add logic to derive FinishedDate from deployment statuses. Preference order: 1. Latest SUCCESS timestamp 2. Latest terminal timestamp (FAILURE/ERROR/INACTIVE/ACTIVE) This improves deployment lifecycle accuracy without modifying existing LatestStatus semantics. * feat(github_graphql): use finished_date in deployment domain conversion Update ConvertDeployment to use tool-layer FinishedDate when populating domain TaskDatesInfo.FinishedDate. Duration is now calculated using resolved finished time instead of UpdatedDate. * chore(github_graphql): add migration for finished_date column Add migration script to auto-migrate GithubDeployment and create nullable finished_date column. * test(github_graphql): update e2e snapshots for deployment finished date and duration Introduce adjustments to existing e2e tests to validate backward compatibility after adding deployment finished date and duration calculation logic. Add `deployment_finished_date` field to `_tool_github_deployments.csv` and `duration_seconds` field to `cicd_deployments.csv`. Ensure historical data remains compatible with the updated behavior. * test(github_graphql): add tests for deployment finished date and duration calculation logic Add test coverage for deployment finished date and duration calculation based on the SUCCESS state instead of relying on the inactive state. Verify fallback behavior when a SUCCESS state is not present. * fix(github_graphql): decouple add finished_date migration from active model Define a dedicated versioned struct for the finished_date migration instead of referencing the evolving GithubDeployment model to ensure deterministic and reproducible schema changes. * chore(github_graphql): fix duplicate import and lint issues in deployment extractor Remove redundant restTasks alias and use githubTasks constants consistently to resolve lint warnings and improve import clarity. * perf(dora): optimize change lead time calculation using batch queries (#8714) * perf(dora): optimize change lead time calculation using batch queries Eliminates N+1 query problem by implementing batch fetching for: - First commits per PR (batchFetchFirstCommits) - First reviews per PR (batchFetchFirstReviews) - Deployments per project (batchFetchDeployments) Performance improvement: - Before: 3 queries per PR (30,001 queries for 10K PRs) - After: 3 batch queries total (99.99% reduction) Also fixes NULL author_id handling in review queries to properly handle PRs with empty author_id field. Tested with E2E tests confirming correctness and performance gains. Signed-off-by: Ankesh <athakur@g2.com> * fix(dora): remove obsolete single-query functions replaced by batch queries Signed-off-by: Ankesh <athakur@g2.com> * ci: add workflow to build and push amd64 image to GHCR Signed-off-by: Ankesh <athakur@g2.com> * revert: remove GHCR build workflow Signed-off-by: Ankesh <athakur@g2.com> --------- Signed-off-by: Ankesh <athakur@g2.com> * Add GitHub Copilot Connection and associated dashboards (#8728) * feat(gh-copilot): add backend plugin * feat(config-ui): add gh-copilot connection UI * feat(grafana): add GitHub Copilot dashboards * fix: gh-copilot CI fixes * use non-admin token for sonarQube (#8731) * fix(devcontainer): make Config UI and Grafana accessible from host when using devcontainers (#8645) * fix(devcontainer): Make config-ui dev server accessible outside the container when started by listening on 0.0.0.0 with the `--host` flag. Also ensure it uses port 4000 or fails to start with `--strictPort` to prevent silent port change that will mismatch with the exported dev container port. * fix(devcontainer): make Config UI and Grafana accessible from host when using devcontainers * feat(github): add username filtering helper for bot exclusion (#8716) * feat(github): add username filtering helper for bot exclusion Implements shouldSkipByUsername() function to filter bot accounts by username using GITHUB_PR_EXCLUDELIST environment variable. - Case-insensitive matching - Comma-separated list support - Whitespace trimming - Returns false for empty usernames or empty exclusion list Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: AvivGuiser <avivguiser@gmail.com> * feat(github): filter bot PRs in extractor Adds username filtering to PR extractor to skip bot-authored PRs when GITHUB_PR_EXCLUDELIST is set. - Checks author username before extraction - Logs debug message when PR is skipped - Includes unit test for bot filtering - Includes e2e test data for bot filtering Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat(github): filter bot reviews in extractor Adds username filtering to review extractor to skip bot reviews when GITHUB_PR_EXCLUDELIST is set. - Checks reviewer username before extraction - Logs debug message when review is skipped - Includes e2e test for bot filtering Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat(github): filter bot PR review comments in extractor Adds username filtering to PR review comment extractor to skip bot comments when GITHUB_PR_EXCLUDELIST is set. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat(github): filter bot issue comments in extractor Adds username filtering to issue comment extractor to skip bot comments when GITHUB_PR_EXCLUDELIST is set. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * docs(github): add bot filtering documentation Documents GITHUB_PR_EXCLUDELIST configuration and usage. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix(github): fix CI errors in bot filtering PR - Add Apache license header to README_FILTERING.md - Export ResetExcludedUsernamesForTest to fix unused lint error - Call ResetExcludedUsernamesForTest in e2e tests to reset sync.Once cache before setting GITHUB_PR_EXCLUDELIST env var Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Signed-off-by: AvivGuiser <avivguiser@gmail.com> Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: auto-enrich missing URL fields in GitHub scopes (#8661) Add automatic URL enrichment to PutScopes function to fetch missing htmlUrl and cloneUrl fields from GitHub API when only fullName is provided. This ensures gitextractor has the required URLs for cloning repositories. - Fetch repo details from GitHub API when URLs are missing - Populate htmlUrl and cloneUrl from API response - Add graceful error handling for inaccessible repos * fix(q_dev): prevent data duplication in user_report and user_data tables (#8737) * fix(q_dev): prevent data duplication in user_report and user_data tables Replace auto-increment ID with composite primary keys so that CreateOrUpdate can properly deduplicate rows on re-extraction. - user_report PK: (connection_id, scope_id, user_id, date, client_type) - user_data PK: (connection_id, scope_id, user_id, date) - Switch db.Create() to db.CreateOrUpdate() in s3_data_extractor - Migration drops old tables, rebuilds with new PKs, resets s3_file_meta processed flag to trigger re-extraction * fix(q_dev): gofmt archived user_data_v2 model * feat(github): Extend exclusion of file extensions to github plugin (#8719) * feat(github): extend PR size exclusion for specified file extension to github plugin * fix: register migration script * fix: move PR size to 'Additional settings' and change so the comma doesn't get removed while typing * fix: linting * fix(doc): update expired Slack invite links in README (#8739) The Slack invite links in README.md were expired and returning "This link is no longer active." Updated both occurrences (badge and community section) to match the current link on the official DevLake website. Closes #8738 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> * docs: add gh-devlake CLI to Getting Started installation options (#8733) Adds gh-devlake as a third installation method alongside Docker Compose and Helm. gh-devlake is a GitHub CLI extension that automates DevLake deployment, configuration, and monitoring from the terminal. Closes #8732 * fix(gitlab): add missing repos scope in project_mapping (#8743) GitLab's makeScopeV200 did not create a repos scope when scopeConfig.Entities was empty or only contained CROSS. This caused project_mapping to have no table='repos' row, breaking downstream DORA metrics, PR-issue linking, and all PR dashboard panels that join on project_mapping. The fix aligns GitLab with the GitHub plugin by: 1. Defaulting empty entities to plugin.DOMAIN_TYPES 2. Adding DOMAIN_TYPE_CROSS to the repo scope condition Closes #8742 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> * fix(grafana): update dashboard descriptions to list all supported data sources (#8741) Several dashboard introduction panels hardcoded "GitHub and Jira" as required data sources, even though the underlying queries use generic domain layer tables that work with any supported Git tool or issue tracker. Updated to list all supported sources following the pattern already used by DORA and WorkLogs dashboards. Closes #8740 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> * fix: modify cicd_deployments name from varchar to text (#8724) * fix: modify cicd_deployments name from varchar to text * fix: update the year * fix(q_dev): replace MariaDB-specific IF NOT EXISTS syntax with DAL methods for MySQL 8.x compatibility (#8745) * fix(azuredevops): default empty entities and add CROSS to repo scope in makeScopeV200 (#8751) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopeV200 produced zero scopes, leaving project_mapping with no rows. Additionally, the repo scope condition did not check for DOMAIN_TYPE_CROSS, so selecting only CROSS would not create a repo scope, breaking DORA metrics. This adds the same fixes applied to GitLab in #8743. Closes #8749 * fix(bitbucket): default empty entities to all domain types in makeScopesV200 (#8750) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopesV200 produced zero scopes, leaving project_mapping with no repo rows. This adds the same empty-entities default applied to GitLab in #8743. Closes #8748 * feat(gh-copilot): add support for organization daily user metrics (#8747) * feat(circleci): add server version requirement and endpoint help text (#8757) Update CircleCI connection form to indicate Server v4.x+ requirement and provide guidance for server endpoint configuration. Signed-off-by: Joshua Smith <jbsmith7741@gmail.com> * feat(asana): add Asana plugin for project and task collection (#8758) Add a new Asana plugin that integrates with Asana's REST API to collect projects, sections, tasks, subtasks, stories (comments), tags, and users, mapping them to DevLake's ticket/board domain model. Backend: - Plugin implementation with all required interfaces (PluginMeta, PluginTask, PluginModel, PluginMigration, PluginSource, PluginApi, DataSourcePluginBlueprintV200) - Collectors, extractors, and converters for projects, sections, tasks, subtasks, stories, tags, and users - Remote API scope picker (Workspaces -> Teams/Portfolios -> Projects) - Scope config with issue-type regex transformation rules - Migration scripts for schema evolution - E2E tests with CSV fixtures for project and task data flows Config UI: - Plugin registration with connection form (PAT auth, endpoint, proxy) - Scope config transformation form for issue-type mapping - Dashboard URL integration for onboarding flow Grafana: - Asana dashboard with task metrics and visualizations Made-with: Cursor * feat: GitHub App token refresh (#8746) * feat(github): auto-refresh GitHub App installation tokens Add transport-level token refresh for GitHub App (AppKey) connections. GitHub App installation tokens expire after ~1 hour; this adds proactive refresh (before expiry) and reactive refresh (on 401) using the existing TokenProvider/RefreshRoundTripper infrastructure. New files: - app_installation_refresh.go: refresh logic + DB persistence - refresh_api_client.go: minimal ApiClient for token refresh POST - cmd/test_refresh/main.go: manual test script for real GitHub Apps Modified: - connection.go: export GetInstallationAccessToken, parse ExpiresAt - token_provider.go: add refreshFn for pluggable refresh strategies - round_tripper.go: document dual Authorization header interaction - api_client.go: wire AppKey connections into refresh infrastructure - Tests updated for new constructors and AppKey refresh flow * feat(github): add diagnostic logging to GitHub App token refresh Add structured logging at key decision points for token refresh: - Token provider creation (connection ID, installation ID, expiry) - Round tripper installation (connection ID, auth method) - Proactive refresh trigger (near-expiry detection) - Refresh start/success/failure (old/new token prefixes, expiry times) - DB persistence success/failure - Reactive 401 refresh and skip-due-to-concurrent-refresh All logs route through the DevLake logger to pipeline log files. * fix(github): prevent deadlock and fix token persistence in App token refresh Deadlock fix: NewAppInstallationTokenProvider now captures client.Transport (the base transport) before wrapping with RefreshRoundTripper. The refresh function uses newRefreshApiClientWithTransport(baseTransport) to POST for new installation tokens, bypassing the RefreshRoundTripper entirely. Token persistence fix: PersistEncryptedTokenColumns() manually encrypts tokens via plugin.Encrypt() then writes ciphertext via dal.UpdateColumns with conn.TableName() (a string) as the first argument. Passing the table name string makes GORM use Table() instead of Model(), preventing the encdec serializer from corrupting the in-memory token value. The encryption secret is threaded from taskCtx.GetConfig(ENCRYPTION_SECRET) through CreateApiClient to TokenProvider to persist functions. Also persists the initial App token at startup for DB consistency, and adds TestProactiveRefreshNoDeadlock with a real RSA key to verify the deadlock scenario is resolved. * fix(grafana): update dashboard descriptions to list all supported data sources (#8741) Several dashboard introduction panels hardcoded "GitHub and Jira" as required data sources, even though the underlying queries use generic domain layer tables that work with any supported Git tool or issue tracker. Updated to list all supported sources following the pattern already used by DORA and WorkLogs dashboards. Closes #8740 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> * fix: modify cicd_deployments name from varchar to text (#8724) * fix: modify cicd_deployments name from varchar to text * fix: update the year * fix(q_dev): replace MariaDB-specific IF NOT EXISTS syntax with DAL methods for MySQL 8.x compatibility (#8745) * fix(azuredevops): default empty entities and add CROSS to repo scope in makeScopeV200 (#8751) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopeV200 produced zero scopes, leaving project_mapping with no rows. Additionally, the repo scope condition did not check for DOMAIN_TYPE_CROSS, so selecting only CROSS would not create a repo scope, breaking DORA metrics. This adds the same fixes applied to GitLab in #8743. Closes #8749 * fix(bitbucket): default empty entities to all domain types in makeScopesV200 (#8750) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopesV200 produced zero scopes, leaving project_mapping with no repo rows. This adds the same empty-entities default applied to GitLab in #8743. Closes #8748 * fix(github): remove unused refresh client constructor and update tests --------- Co-authored-by: Spiff Azeta <35563797+spiffaz@users.noreply.github.com> Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> Co-authored-by: Dan Crews <crewsd@gmail.com> Co-authored-by: Tomoya Kawaguchi <68677002+yamoyamoto@users.noreply.github.com> * fix: cwe89 sql injection (#8762) * feat(q-dev): add logging data ingestion and enrich Kiro dashboards (#8767) * feat(q-dev): add logging data ingestion and enrich Kiro dashboards Add support for ingesting S3 logging data (GenerateAssistantResponse and GenerateCompletions events) into new database tables, and enrich all three Kiro Grafana dashboards with additional metrics. Changes: - New models: QDevChatLog and QDevCompletionLog for logging event data - New extractor: s3_logging_extractor.go parses JSON.gz logging files - Updated S3 collector to also handle .json.gz files - Added logging S3 prefixes (GenerateAssistantResponse, GenerateCompletions) - New dashboard: "Kiro AI Activity Insights" with 10 panels including model usage distribution, active hours, conversation depth, feature adoption (Steering/Spec), file type usage, and prompt/response trends - Enriched "Kiro Code Metrics Dashboard" with DocGeneration, TestGeneration, and Dev (Agentic) metric panels - Fixed "Kiro Usage Dashboard" per-user table to sort by user_id - Migration script for new tables * fix(q-dev): use separate base path for logging S3 prefixes Logging data lives under a different S3 prefix ("logging/") than user report data ("user-report/"). Add LoggingBasePath option (defaults to "logging") so logging prefixes are constructed correctly. * fix(q-dev): auto-scan logging path without extra config Kiro exports to two well-known S3 prefixes in the same bucket: - user-report/AWSLogs/{accountId}/KiroLogs/ (CSV reports) - logging/AWSLogs/{accountId}/KiroLogs/ (interaction logs) When AccountId is set, automatically scan both paths. The "logging" prefix is hardcoded since it's a standard Kiro export convention. No additional configuration needed. * fix(q-dev): update scope tooltip to mention logging data scanning * fix(q-dev): fix scope ID routing and CSV/JSON file separation Three fixes: 1. Use *scopeId (catch-all) route pattern instead of :scopeId so scope IDs containing "/" (e.g. "034362076319/2026") work in URL paths 2. CSV extractor now filters for .csv files only, preventing it from trying to parse .json.gz logging files as CSV 3. Frontend scope API calls now encodeURIComponent(scopeId) for safe URL encoding * fix(q-dev): resolve *scopeId route conflict with dispatcher pattern The catch-all *scopeId route conflicts with *scopeId/latest-sync-state. Follow Jenkins/Bitbucket pattern: use a single *scopeId route with a GetScopeDispatcher that checks for /latest-sync-state suffix and dispatches accordingly. All scope handlers now TrimLeft "/" from scopeId. * fix(q-dev): use URL-safe scope ID format (underscore separator) Scope IDs like "034362076319/2026" break URL routing because "/" is a path separator. Change ID format to "034362076319_2026" (underscore) when AccountId is set. The Prefix field still uses "/" for S3 path matching. Revert to standard :scopeId routes since IDs are now safe. Note: existing scopes need to be recreated after this change. * fix(q-dev): use NoPKModel instead of Model in archived logging models archived.Model only has ID+timestamps, missing RawDataOrigin fields (_raw_data_params etc.) that common.NoPKModel includes. This caused "Unknown column '_raw_data_params'" errors at runtime. * fix(q-dev): fix GROUP BY in per-user table to merge display_name variants Remove display_name from GROUP BY so same user_id with different display_name values gets merged. Use MAX(display_name) in SELECT. * fix(q-dev): normalize logging user IDs to match CSV short UUID format Logging data uses "d-{directoryId}.{UUID}" format while CSV user-report uses plain "{UUID}". Strip the "d-xxx." prefix so the same user maps to one user_id across both data sources. * fix(q-dev): normalize user IDs in CSV extractors and sort table DESC Apply normalizeUserId to both createUserReportData and createUserDataWithDisplayName so user_report CSV data also strips the "d-{directoryId}." prefix. Change per-user table sort to ORDER BY user_id DESC. * style(q-dev): fix gofmt formatting in chat_log models * perf(q-dev): parallelize logging S3 downloads and batch DB writes Optimize logging extractor performance: - 10 goroutine workers for parallel S3 file downloads - Batch 50 files per DB transaction instead of 1-per-file - sync.Map cache for display name resolution (avoid repeated IAM calls) - Parse records in memory during download, write all at once This should improve throughput from ~1.5 files/sec to ~15+ files/sec for typical logging file sizes. * fix(q-dev): check tx.Rollback error return to satisfy errcheck lint * feat(q-dev): add per-user model usage table and models column Add "Per-User Model Usage" table (panel 11) showing each user's request count and avg prompt/response length per model_id. Also add "Models Used" column to the Per-User Activity table. * fix(q-dev): remove per-user model usage table, keep models column only * feat(q-dev): add Kiro Executive Dashboard with cross-source analytics New dashboard "Kiro Executive Dashboard" with 12 panels covering: - KPIs: WAU, credits efficiency, acceptance rate, steering adoption - Trends: weekly active users, new vs returning users - Adoption funnel: Chat→Inline→CodeFix→Review→DocGen→TestGen→Agentic→Steering→Spec - Cost: credits pace vs projected monthly, idle power users - Quality: acceptance rate trends, code review findings, test generation - Efficiency: per-user productivity table with credits/line ratio Correlates data across user_report (credits), user_data (code metrics), and chat_log (interaction patterns) for holistic Kiro usage insights. * fix(q-dev): fix pie charts to show per-row slices instead of single total Set reduceOptions.values=true so Grafana treats each SQL result row as a separate pie slice. Fixes Model Usage Distribution, File Type Usage, Kiro Feature Adoption, and Active File Types pie charts. * fix(q-dev): cast Hour to string for Active Hours bar chart x-axis * fix(q-dev): fix pie chart single-slice and GROUP BY display_name issues 1. qdev_user_report Panel 4 (Subscription Tier Distribution): set reduceOptions.values=true to show per-tier slices 2. qdev_user_data Panel 6 (User Interactions): remove display_name from GROUP BY, use MAX(display_name) to merge same user * fix(q-dev): prevent data inflation in user_report JOIN user_data user_report has multiple rows per (user_id, date) due to client_type (KIRO_IDE, KIRO_CLI), but user_data has only one row per (user_id, date). A direct JOIN causes user_data metrics to be counted multiple times. Fix: pre-aggregate user_report by (user_id, date) in a subquery before joining, so the JOIN is always 1:1. Affects: Credits Efficiency stat and User Productivity table. * feat(qa): add is_invalid field to qa_test_case_executions (#8764) * feat(qa): add is_invalid field to qa_test_case_executions Add is_invalid boolean field to the domain layer qa_test_case_executions table to allow QA teams to flag test executions as invalid due to environmental issues, flaky tests, false positives, or false negatives. Changes: - Add IsInvalid field to QaTestCaseExecution domain model - Create migration script (20260313_add_is_invalid_to_qa_test_case_executions) - Register migration in migrationscripts/register.go - Update customize service to set default value for is_invalid - Update E2E test data to include new column Resolves #8763 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(qa): handle missing is_invalid column in CSV import Fix PostgreSQL compatibility issue when CSV files don't contain the is_invalid column. The field now defaults to false instead of an empty string. Changes: - Update qaTestCaseExecutionHandler to check for empty string values - Add E2E test for backward compatibility with CSV files lacking is_invalid - Add explicit IsInvalid initialization in Testmo plugin converter Resolves #8763 --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat(linker): link when branch names contain issue keys (#8777) * feat(linker): branch names containing issue keys * chore: add testing data * Add codespell support with configuration and fixes (#8761) * ci(codespell): add codespell config and GitHub Actions workflow Add .codespellrc with skip patterns for generated files, camelCase/PascalCase ignore-regex, and project-specific word list (convertor, crypted, te, thur). Add GitHub Actions workflow to run codespell on push to main and PRs. Co-Authored-By: Claude Code 2.1.63 / Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Yaroslav Halchenko <debian@onerussian.com> * fix(codespell): fix ambiguous typos requiring context review Manual fixes for typos that needed human review to avoid breaking code: - Comment/string typos: occured->occurred, destory->destroy, writting->writing, retreive->retrieve, identifer->identifier, etc. - Struct field comments and documentation corrections - Migration script comment fixes (preserving Go identifiers like DataConvertor) Co-Authored-By: Claude Code 2.1.63 / Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Yaroslav Halchenko <debian@onerussian.com> * fix(codespell): fix non-ambiguous typos with codespell -w Automated fix via `codespell -w` for clear-cut typos across backend, config-ui, and grafana dashboards. Examples: sucess->success, occurence->occurrence, exeucte->execute, asynchornous->asynchronous, Grafana panel typos, etc. Co-Authored-By: Claude Code 2.1.63 / Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Yaroslav Halchenko <debian@onerussian.com> --------- Signed-off-by: Yaroslav Halchenko <debian@onerussian.com> Co-authored-by: Claude Code 2.1.63 / Claude Opus 4.6 <noreply@anthropic.com> * feat(q-dev): enrich logging fields, separate dashboards, add E2E tests (#8786) * feat(q-dev): enrich logging fields, separate dashboards by data source, add E2E tests - Add new fields to chat_log: CodeReferenceCount, WebLinkCount, HasFollowupPrompts (from codeReferenceEvents, supplementaryWebLinksEvent, followupPrompts in JSON) - Add new fields to completion_log: LeftContextLength, RightContextLength (from leftContext/rightContext in JSON) - Update s3_logging_extractor to parse and populate new fields - Add migration script 20260319_add_logging_fields - Create qdev_feature_metrics dashboard for legacy by_user_analytic data - Reorganize qdev_executive dashboard with Row dividers labeling data sources and cross-dashboard navigation links - Enrich qdev_logging dashboard with new panels: Chat Trigger Type Distribution, Response Enrichment Breakdown, Completion Context Size Trends, Response Enrichment Trends - Fix SQL compatibility with only_full_group_by mode in executive dashboard (Weekly Active Users Trend, New vs Returning Users) - Fix Steering Adoption stat panel returning string instead of numeric value - Add Playwright E2E test covering full pipeline flow and dashboard verification * fix: add Apache license headers to e2e files, fix gofmt alignment * fix: add SQL identifier validation to prevent SQL injection via table/column names (#8769) Add ValidateTableName and ValidateColumnName functions in core/dal to ensure table and column names used in dynamic SQL are safe identifiers. Applied to scope_service_helper, scope_generic_helper, and customized_fields_extractor. * feat(q-dev): add Kiro Credits + DORA Correlation dashboard (#8792) Add a new Grafana dashboard that correlates Kiro AI usage (credits, messages, active users) with DORA metrics at weekly aggregate level. Panels include: - Pearson's r correlation between weekly credits and PR cycle time - High AI Usage vs Low AI Usage cycle time comparison - Weekly credits vs deployment frequency trend - Weekly credits vs change failure rate trend Data is joined by week_start between _tool_q_dev_user_report and project_pr_metrics / cicd_deployment_commits. * feat(q-dev): add AI Cost-Efficiency dashboard (#8793) Add a Grafana dashboard showing AI tool cost-efficiency metrics: - Credits per merged PR (overall + weekly trend) - Credits per production deployment (overall + weekly trend) - Credits per issue resolved (overall + weekly trend) - Weekly AI activity volume (credits, messages, conversations) Joins _tool_q_dev_user_report with pull_requests, cicd_deployment_commits, and issues by weekly aggregation. * feat(q-dev): add Multi-AI Tool Comparison dashboard (Copilot vs Kiro) (#8794) Add a Grafana dashboard comparing GitHub Copilot and Kiro side by side: - Weekly active users comparison - Code suggestions & acceptance events (per tool) - LOC accepted comparison (combined time series) - Acceptance rate comparison (bar gauge) Template variables for Copilot connection/scope selection. Data from _tool_copilot_enterprise_daily_metrics vs _tool_q_dev_user_report and _tool_q_dev_user_data. * feat(q-dev): add Kiro AI Model ROI dashboard (#8795) Add a Grafana dashboard analyzing per-model performance from chat logs: - Model Performance Summary table (requests, share%, avg prompt/response length, response/prompt ratio, steering/spec mode usage) - Daily Model Usage Distribution (stacked bar chart) - Avg Response Length by Model trend (output quality proxy) Data source: _tool_q_dev_chat_log grouped by model_id. * feat(q-dev): add Steering & Spec Mode Adoption dashboard (#8798) Track Kiro steering rules and spec mode adoption: - User/request adoption rate stats - Weekly adoption rate trend - Steering impact on prompt/response length - Per-user feature adoption table * feat(q-dev): add Developer AI Productivity Hours dashboard (#8797) Analyze when developers are most productive with AI tools: - AI Activity by Hour of Day (chat + completions stacked bar) - Prompt & Response Length by Hour (complexity patterns) - Feature Usage by Hour (steering/spec mode/plain chat) - AI Activity by Day of Week * feat(q-dev): add Language AI Heatmap dashboard (#8796) Analyze AI-assisted coding patterns by programming language: - Language Completion Profile table (requests, avg completions, context sizes, users per language) - Daily Completions by Language (stacked bar) - Active File Types During Chat (donut) - Avg Context Size by Language trend (top 5) * Fix/circleci column names (#8799) * fix(circleci): rename created_at to created_date in jobs/workflows Add migration to copy created_at -> created_date and update models/converters. * fix(circleci): update pipeline parsing * test(circleci): add incremental tests for collectors * fix(jenkins): scope multi-branch build collection to current project (#8430) (#8781) The branch jobs query in collectMultiBranchJobApiBuilds selected all WorkflowJob entries across all multi-branch pipelines for a connection, causing builds to be duplicated and misattributed. Filter by _raw_data_params to collect only the current project's branch jobs. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: Make gh-copilot plugin database agnostic (#8779) Co-authored-by: Eldrick Wega <eldrick.wega@outlook.com> * fix(sonarqube): increase cq_issues and cq_file_metrics project_key length to 500 (#8783) Fixes #8331 * feat: added taiga plugin (#8755) * feat: added taiga plugin * fix: fixed tests * feat(gh-copilot): add support for organization daily user metrics (#8747) * feat(circleci): add server version requirement and endpoint help text (#8757) Update CircleCI connection form to indicate Server v4.x+ requirement and provide guidance for server endpoint configuration. Signed-off-by: Joshua Smith <jbsmith7741@gmail.com> * fix: fixed test files --------- Signed-off-by: Joshua Smith <jbsmith7741@gmail.com> Co-authored-by: Reece Ward <47779818+ReeceXW@users.noreply.github.com> Co-authored-by: Joshua Smith <jbsmith7741@gmail.com> * fix(gh-copilot): skip empty report responses for dates without Copilot data (#8804) GitHub's Copilot Usage Metrics Reports API returns HTTP 200 with body "" for dates before usage data was available, instead of returning a 404. The existing ignore404 callback does not catch this, so the ResponseParser attempts json.Unmarshal on "" which fails and crashes the collector pipeline. Add an isEmptyReport guard that detects empty/null bodies and returns nil so the collector silently skips those days. Fixes apache#8789 * fix(docker): pin Poetry to 2.2.1 for Python 3.9 compatibility (#8735) Poetry 2.3.0 dropped Python 3.9 support. Without cache the installer fetches the latest version (currently 2.3.2), which fails on the python:3.9-slim-bookworm base image. Pin to 2.2.1, the last release compatible with Python 3.9. Co-authored-by: Rodrigo Silva <rodrigo.silva@bonial.com> * fix(linker): scope clearHistoryData to current project only (#8814) (#8815) The clearHistoryData() function used a LEFT JOIN with project_name in the ON clause, causing the subquery to return all PR IDs regardless of project. This effectively wiped the entire pull_request_issues table on every linker run, deleting links from other projects sharing the same repos and links created by the GitHub converter. Fix: - Use INNER JOIN + WHERE for proper project scoping - Add issue-side subquery scoped to current project's boards - Filter by _raw_data_table/_raw_data_remark to only delete linker-created rows Add e2e test for cross-project shared repo scenario. * fix(circleci): prevent negative values when calculating circleci (#8800) workflow duration * fix: sonarqube: missing api/users/search endpoint (#8813) * fix(argocd): extract revision from multi-source application revisions[] (#8810) * Fix/gh copilot apiversion (#8826) * fix: changed apiversion of gh copilot as per the docs * fix: added test and a const for testable version * fix(github_graphql): remove stale local issue rows when an issue no longer resolves (#8820) * fix(github_graphql): remove stale local issue rows * fix(github_graphql): prevent stale issue cleanup from crossing repository boundaries The previous missing-issue cleanup only keyed deletes by connection and issue identifiers, which could reach too broadly when an issue was transferred between repositories under the same DevLake connection. This change scopes cleanup to the source repository by carrying RepoId through the refresh path and requiring the source RawDataOrigin parameters before deleting stale tool rows. Constraint: Keep the fix inside the existing github_graphql collector path without schema changes Rejected: Continue deleting by connection_id + github_id only | could remove destination-repository rows after transfers Rejected: Disable stale cleanup for all missing issues | would leave deleted-issue failures recurring Confidence: medium Scope-risk: moderate Directive: Preserve source-scope guards if future cleanup paths touch transferred issues again Tested: go test ./plugins/github_graphql/tasks; go build ./helpers/pluginhelper/api ./plugins/github_graphql/tasks Not-tested: End-to-end transfer scenario against a live GitHub repository * fix: throwing on parsing for a 204 response (#8825) * fix(asana): use float8 instead of double for PostgreSQL compatibility (#8837) The migration used gorm:"type:double" which is MySQL-specific. PostgreSQL does not have a "double" type and fails with: ERROR: type "double" does not exist (SQLSTATE 42704) Changed to float8 which works on both MySQL (alias for double) and PostgreSQL (alias for double precision). Closes #8835 Co-authored-by: tengtian <tengtianmoemoe@gamil,com> * Fix/8787 cancel pipeline keeps running (#8832) * fix(pipeline): cancel pipeline returns 200 OK but keeps running (#8787) Three independent bugs caused pipeline cancellation to silently fail: 1. CancelPipeline discarded errors from CancelTask and never cancelled TASK_CREATED tasks in future stages. Now running tasks are cancelled via context, non-running tasks are batch-updated to TASK_CANCELLED in the DB, and errors are logged and returned. 2. gitextractor's storeRepoSnapshot (go-git path) had no ctx.Done() checks in its commit/blame loops, making it unresponsive to cancellation for 30+ minutes on large repos. Added cancellation checkpoints following the pattern already used elsewhere in the file. 3. Cancelled tasks were marked TASK_FAILED instead of TASK_CANCELLED, and ComputePipelineStatus never returned TASK_CANCELLED. Now RunTask checks for context cancellation and writes TASK_CANCELLED, and ComputePipelineStatus returns TASK_CANCELLED when the pipeline was cancelled by the user. Test gaps: RunTask deferred status logic, CancelPipeline flow, and storeRepoSnapshot have no existing unit tests. These are pre-existing gaps not introduced by this change. The only existing test, TestComputePipelineStatus, has been extended to cover isCancelled=true. Closes #8787 Related: #5585, #4188 * refactor(pipeline): improve CancelPipeline readability and correctness Extract cancelRunningTasks and cancelPendingTasksInDB helpers from CancelPipeline for better separation of concerns. Also fixes: - .As(NotFound) replaced with .GetType() to prevent swallowing wrapped errors - Added TASK_RESUME to cancelPendingTasksInDB status filter - Error message now includes pipeline ID for traceability - Added TestCancelPipeline e2e tests with 3 subtests * fix(bitbucket): handle microsecond precision timestamps in PR extractor (#8828) Bitbucket Cloud returns timestamps with 6-digit fractional seconds (e.g. 2025-09-15T08:53:36.337932+00:00). Changed BitbucketCreatedAt/ BitbucketUpdatedAt from time.Time to *common.Iso8601Time and added microsecond format to DateTimeFormats. Fixes #8708 Co-authored-by: Vincenzo Baldi <vincenzobaldi@a-m-j-v-baldi.fritz.box> * fix(asana): remove MySQL-specific gorm type tags for PostgreSQL compatibility (#8836) Remove explicit `gorm:"type:double"` and `gorm:"type:int unsigned"` tags from the Asana task migration struct. PostgreSQL does not have a `double` type (it uses `double precision`) and does not support `UNSIGNED` integer qualifiers. Without explicit tags, GORM infers the correct column type per dialect automatically. Follows the same pattern as #8779 which fixed equivalent issues in the copilot plugin. closes #8835 Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Klesh Wong <klesh@qq.com> * fix(docker): Add BUILDPLATFORM to config-ui builder stage and update node version (#8845) - Use BUILDPLATFORM so the Node builder runs natively on the build host (e.g. arm64 on Apple Silicon) rather than under x64 emulation which can cause crashes - Node 22 LTS builder replaces v18 which reached EOL in April 2025 * fix(github): bisect adaptive time windows for workflow runs 40k pagination cap (#8844) GitHub's /actions/runs enforces a 40k cap in unfiltered mode and a 1,000-item cap per filtered search, making any repo with >40k workflow runs uncollectable. Switch to filtered mode and recursively bisect time windows at integer-second midpoints, probing via SubmitBlocking to share rate-limit with the main collector and feeding leaves to a single ApiCollector so raw-table Delete fires only once. See PR description for the full design rationale. Closes #8842 Signed-off-by: yamoyamoto <yamo7yamoto@gmail.com> * fix(sonarqube): use filter parameter for project search (#8849) SonarQube's components/search_projects endpoint does not accept a bare q parameter for text search; it is silently ignored. The correct approach is the filter language: filter=query = "term". Sending q=term caused DevLake to return the full unfiltered project list regardless of the search input. Also fix three related pagination bugs in the remote scope search UI: - results were replaced instead of accumulated on page 2+ - getHasMore was inverted (tested loading state instead of hasMore) - onScroll could trigger duplicate requests while one was in flight - typing a new query did not reset page and hasMore state Fixes #8805 Signed-off-by: Joshua Smith <jbsmith7741@gmail.com> * fix: disable API client caching for GitHub App connections (#8850) GitHub App installation tokens expire after ~1 hour, causing admin APIs (remote-scopes, test-connection) to fail silently after token expiry. This fix overrides GetHash() to return empty string for GitHub App connections, disabling client caching and ensuring fresh tokens are fetched for each admin request. PAT connections continue to use default caching behavior. Fixes #8847 * fix(ci): pin docker/* actions in build.yml to ASF-approved SHAs (#8852) Signed-off-by: yamoyamoto <yamo7yamoto@gmail.com> * feat(auth): implement oidc authentication and session management (#8854) DevLake had no native OIDC login. The existing protections: nginx Basic auth and /rest/* Bearer tokens, all sit outside the API and don't give you SSO at the application layer. This adds an OIDC login flow behind a default-off AUTH_ENABLED switch. Multiple providers (Entra ID, Google, etc.) can be configured side by side, sessions are revocable server-side, and CSRF is enforced on state-changing requests when authenticating via the session cookie. When AUTH_ENABLED=false the system behaves exactly as before; existing Basic auth, oauth2-proxy, and API-key flows are untouched. Refs: https://github.com/apache/incubator-devlake/discussions/8694 Refs: https://devlake-io.slack.com/archives/C03AJH1HBAT/p1776917914734679 * fix(github_graphql): keep requestedIssues out of GraphQL query in Collect Issues (#8855) Signed-off-by: yamoyamoto <yamo7yamoto@gmail.com> * fix(server): use interface{} for store handler swag annotations (#8859) Signed-off-by: yamoyamoto <yamo7yamoto@gmail.com> * fix(argoCD): avoid duplication of latest deployments (#8862) * fix(azuredevops_go): handle empty/invalid timeline responses for YAML… (#8839) * fix(azuredevops_go): handle empty/invalid timeline responses for YAML-broken builds Builds that fail due to YAML syntax errors in pipeline definitions return an empty or non-JSON body from the Timeline API (200 OK with no usable content). Previously this caused ParseRawMessageFromRecords to propagate a hard error that aborted the entire collectApiTimelineRecords subtask. Two complementary fixes: 1. Add ignoreInvalidTimelineResponse AfterResponse handler that supersedes ignoreDeletedBuilds: it still skips 404s (deleted builds) and now also skips responses with empty or non-JSON bodies by returning api.ErrIgnoreAndContinue, leaving the rest of the pipeline intact. The body is peeked and then restored via io.NopCloser so the downstream ResponseParser can still read it when the body is valid JSON. 2. Extend the DB query in CollectRecords to exclude builds with result = 'none' in addition to result = 'failed'. Builds broken by YAML syntax errors are categorised as 'none' by Azure DevOps (already mapped to RESULT_FAILURE in cicdBuildResultRule), so fetching their timeline is both unnecessary and the primary source of the bad responses. Using NOT IN ? with a []string slice follows the same pattern used in blueprint_helper.go. Unit tests added for ignoreInvalidTimelineResponse covering: 404, empty body, non-JSON body, valid JSON (nil error + body restored), and valid empty JSON object. Fixes: #8838 Signed-off-by: Cornelius Schuchardt <cornelius.schuchardt@beck.de> * fix(azuredevops_go): remove unused ignoreDeletedBuilds function The ignoreDeletedBuilds function was replaced by ignoreInvalidTimelineResponse in ci_cd_timeline_records_collector.go but was left behind in shared.go. This caused the 'unused' linter to fail with exit code 2. * fix: resolve govet non-constant format string warnings Use constant format verbs instead of dynamic strings passed directly as format arguments to avoid govet printf-family warnings: - core/migration/linter: fmt.Errorf with string concat replaced by %s verb - impls/logruslog: Warn/Error pass pre-formatted message via "%s" arg instead of using it directly as the format string --------- Signed-off-by: Cornelius Schuchardt <cornelius.schuchardt@beck.de> * feat(ui): implement dark theme and refactor color usage (#8865) Add a system-aware dark theme with a header toggle (light, dark, system) that persists in localStorage. All colors now flow through `src/theme/tokens.ts` into both the antd component library and styled-components, replacing hardcoded hex across `styled.ts` files and inline styles. Light mode is preserved the same as before. https://github.com/apache/incubator-devlake/issues/8863 * feat(github): Add incremental data collection (#8858) * fix(github): use RFC 3339 format for since= incremental API parameters The since= query parameter passed to the GitHub API in four collectors was formatted using Go's time.Time.String(), which produces a human-readable string (e.g. "2024-01-15 10:30:00 +0000 UTC") rather than the ISO 8601 / RFC 3339 format the GitHub API requires (e.g. "2024-01-15T10:30:00Z"). The GitHub API silently ignores malformed date strings, causing these collectors to perform full re-scans on every incremental run despite appearing to filter correctly. Fix by using .UTC().Format(time.RFC3339) in all four affected collectors: - comment_collector.go - issue_collector.go - commit_collector.go - pr_review_comment_collector.go (cherry picked from commit cf3cb621462d8ae661df5fcb8e1b47c70564cd60) * feat(github): migrate event extractor to StatefulApiExtractor Switch Extract Events from the legacy full-scan NewApiExtractor to NewStatefulApiExtractor, which filters _raw_github_api_events by created_at >= last_run_start on incremental syncs. This table had 13,828 rows in a representative production run and took ~117s to process on every run regardless of how few new events were collected. After this change incremental runs process only newly collected rows. (cherry picked from commit 84f324f65e8576cd2ff0ca8b74a2588ee2600e12) * feat(github): migrate PR extractor to StatefulApiExtractor Switch Extract Pull Requests from legacy full-scan NewApiExtractor to NewStatefulApiExtractor. The _raw_github_api_pull_requests table had 12,448 rows in a representative production run, taking ~108s on every incremental sync. After this change only newly collected PR rows are processed. SubtaskConfig captures prType and prComponent regex strings so that a scope config change automatically triggers a full re-extract. BeforeExtract deletes GithubPrLabel rows for the current PR before re-inserting them in incremental mode, preventing stale labels from persisting when labels are removed upstream. (cherry picked from commit 80dc76d4e5f1ad6c65d48d42f60f50b87c9dad2d) * feat(github): migrate workflow run and PR commit extractors to StatefulApiExtractor Switch Extract Workflow Runs (~7,364 raw rows, ~65s) and Extract PR Commits (~6,982 raw rows, ~60s) from legacy full-scan NewApiExtractor to NewStatefulApiExtractor. Both extractors are simple mappings with no scope-config dependency, so no SubtaskConfig or BeforeExtract needed. (cherry picked from commit ea4f158a65ad0f7a2c23ba7bbc9932059e2ca408) * feat(github): migrate remaining high-volume extractors to StatefulApiExtractor Migrate Extract Jobs (~5,369 rows, ~48s), Extract PR Reviews (~3,073 rows, ~27s), and Extract PR Review Comments (~1,820 rows, ~16s) from legacy full-scan NewApiExtractor to NewStatefulApiExtractor. Also moves prUrlRegex compilation in pr_review_comment_extractor.go from inside the Extract closure (recompiled on every raw row) to before the extractor is created, eliminating redundant regexp compilation. (cherry picked from commit 54d601587bc74ebd0f1103345c545571146739b5) * feat(github): migrate remaining low-volume extractors to StatefulApiExtractor Migrate the final seven GitHub extractors to NewStatefulApiExtractor: issue, comment, account, account_org, milestone, commit, commit_stats. issue_extractor gains SubtaskConfig (issue classification regex strings) so scope config changes trigger automatic full re-extraction, and BeforeExtract cleanup for GithubIssueLabel and GithubIssueAssignee rows in incremental mode to prevent stale labels/assignees persisting after upstream removal. All other extractors in this commit are simple migrations with no config-sensitivity or child record cleanup needed. With this commit all 14 GitHub plugin extractors are now incremental. Combined with the collector fixes in earlier commits, incremental collection runs that previously took 9+ minutes in the extract phase will now complete in seconds when few or no new records were collected. (cherry picked from commit 606acea88caef63662733162faa47a6c6d3155cc) * feat(github): migrate CICD converters to StatefulDataConverter Convert Workflow Runs and Convert Jobs now use NewStatefulDataConverter, skipping records unchanged since last run on incremental pipelines. Jobs are filtered via JOIN on _tool_github_runs.github_updated_at. (cherry picked from commit 75a909efccab1d04bfae4058f4674663b00762a6) * feat(github): migrate PR supporting data converters to StatefulDataConverter Convert PR Commits, Convert PR Comments, and Convert PR Reviews now use NewStatefulDataConverter. Child-of-PR records are filtered incrementally via JOIN on _tool_github_pull_requests.github_updated_at; PR comments additionally filter on their own github_updated_at. (cherry picked from commit 17cbc2e4caff7349fa2d3389d7cafcc5d0edee71) * feat(github): migrate PR main and cross converters to StatefulDataConverter Convert Pull Requests filters by GithubPullRequest.github_updated_at. Convert Reviews and Convert PR Issues filter via JOIN on pull_requests github_updated_at since reviewers and pr_issues have no own timestamp. (cherry picked from commit 47437ae49d30b967b443c964a4b1baca34344acc) * feat(github): migrate remaining data converters to StatefulDataConverter Migrate the last 9 converters from DataConverter to StatefulDataConverter so they skip already-processed records on incremental runs: - issue_convertor: filter on github_updated_at - issue_comment_convertor: filter on github_updated_at - issue_label_convertor: JOIN to issues, filter on issues.github_updated_at - issue_assignee_convertor: JOIN to issues, filter on issues.github_updated_at - pr_label_convertor: JOIN to pull_requests, filter on pr.github_updated_at - account_convertor: filter on updated_at - release_convertor: filter on updated_at - repo_convertor: filter on updated_at; retain GithubApiRepo struct (used by pr_extractor) - commit_convertor: filter on authored_date (cherry picked from commit 7bcc9da4676d34d9cd24e41a68765cf85723ac81) * fix(github): keep incremental commit and PR-issue processing accurate * fix(github): bootstrap workflow runs incremental window from tool state * feat(runner): include processed record count in subtask finish logs * fix(stateful): bootstrap subtask state from collector checkpoints * fix(github): optimize incremental Convert Jobs query * fix(github): resolve lint and staticcheck issues in converters * fix(stateful): tolerate missing collector state table in bootstrap * fix(github): address CI failures in state bootstrap and run tests * fix(server): use interface{} for store handler swag annotations (#8859) Signed-off-by: yamoyamoto <yamo7yamoto@gmail.com> * fix(github): stabilize e2e state bootstrap and issue assignee join * fix(stateful): skip collector bootstrap lookup when state table is absent * test(stateful): use generic scope params in bootstrap tests * fix(github): keep issue assignee conversion full-sync safe --------- Signed-off-by: yamoyamoto <yamo7yamoto@gmail.com> Co-authored-by: Tomoya Kawaguchi <68677002+yamoyamoto@users.noreply.github.com> * fix(ipc): increase scanner buffer size to 5MB to handle large outputs (#8841) * fix(ci): replace Poetry runtime setup with uv (#8830) Co-authored-by: Klesh Wong <klesh@qq.com> * feat(docker): add Dockerfile.local for local development and fix migrations (#8866) * fix(migrations): add new fields and improve migration scripts for various plugins * feat(docker): add Dockerfile.local for local development and update docker-compose configuration * chore: add license to dockerfile --------- Co-authored-by: Fábio Luciano <fabio.gois@sicoob.com.br> * feat(rootly): add rootly integration plugin (#8877) * feat(rootly): scaffold plugin skeleton Add the plugin entry point, impl shell with all required plugin interfaces except DataSourcePluginBlueprintV200 (wired in U2), connection model with Bearer auth, service + scope-config + placeholder incident/user/assignment models, and the initial migration script. API resources and subtasks are intentionally empty and will be populated by U2-U5. * feat(rootly): add API handlers and blueprint v200 Wire up connection test, remote scope list/search, scope CRUD, scope sync state, and blueprint v200 endpoints. Remote scope listing speaks JSON:API and page-based pagination to match the Rootly API shape. TestConnection validates the bearer token via GET /users/current. impl.go now wires api.Init and restores DataSourcePluginBlueprintV200 so the plugin can produce pipeline plans for selected Rootly services. * feat(rootly): collect and convert services to domain boards Add services_collector, services_extractor, and service_converter subtasks. Collector fetches GET /services/{id} for the scoped service, unwraps the JSON:API envelope, and populates _tool_rootly_services. Converter emits one ticket.Board per service using the standard domain-id generator, feeding into the existing TICKET pipeline. Also harden remote-scope pagination termination to key off the page we requested rather than meta.current_page so a response missing that field does not silently truncate the service list. * feat(rootly): collect and extract incidents with inline users Finalize the Incident and User models and add role-specific user-id fields (creator, started_by, mitigated_by, resolved_by, closed_by) directly on the Incident row. Add the incidents_collector and incidents_extractor subtasks. The collector is single-phase, filter[services]-scoped and filter[updated_at][gt]-incremental; the extractor unwraps the JSON:API envelope and pulls inline nested user objects from the incident attributes, emitting deduplicated User rows per incident. Drop the Assignment table entirely. Rootly's incident data model is role-per-lifecycle-event, not a list of assignees, so PagerDuty's n-assignees shape does not fit. The schema and GetTablesInfo are adjusted accordingly. * feat(rootly): convert incidents to domain issues Add incidents_converter. For each tool-layer incident, emit a ticket.Issue (Type=INCIDENT) with status, priority, lead time, and resolution date derived from the corresponding Rootly fields; emit one ticket.IssueAssignee per distinct role user (creator, started_by, mitigated_by, resolved_by, closed_by); and emit a ticket.BoardIssue linking the incident to its service board. This feeds DevLake's existing DORA pipeline so change-failure rate and MTTR compute correctly for teams running on Rootly. Unknown incident statuses fall back to IN_PROGRESS with a warning log rather than panicking (a deliberate divergence from the PagerDuty plugin, motivated by Rootly's more volatile status enum). Severity mapping accepts case-insensitive sev0-sev4; unknown severities are preserved as-is. Guard computeLeadTime against resolved-before-started timestamps (clock skew or backfill anomalies) by returning nil rather than the wraparound garbage a naive uint() cast would produce. Tighten test coverage on the dedup, key-fallback, and known-status-warning boundaries flagged during code review. * test(rootly): add e2e test for extract and convert pipeline Add an end-to-end test that drives the extract and convert subtasks over a crafted raw-incident fixture and verifies the tool-layer and domain-layer output against snapshot CSVs. Fixtures cover every branch of the status and severity mapping tables, the same-user-across-roles dedup path, the zero-user case, and the safety-net filter that drops incidents whose relationships point at a different service than the one the task was scoped to. * feat(rootly): register plugin in backend and config-ui Add rootly to the backend plugin startup test and the table-info test so CI exercises it alongside the other plugins. Register RootlyConfig in the config-ui plugin list so operators can create connections, browse scopes, and run blueprints through the UI. The cloud endpoint default includes the /v1/ API version prefix, which is what Rootly's REST API actually expects; without it requests land on the marketing site and return 404. The DOC_URL entries point at a Configuration/Rootly docs page that still needs to be written. Use the Rootly wordmark glyph as the plugin icon, rewritten as a fill-aware (currentColor) SVG so the config-ui can recolor it for selected/unselected states the same way it does every other plugin icon. * fix(rootly): align archived models with live model schemas The U1 init migration's archived Connection, Service, and ScopeConfig models were missing columns contributed by the live models' embedded helpers, so AutoMigrate produced tables the live models could not read or write. Each gap surfaced as an "Unknown column" error from MySQL the first time the table was touched: - Connection was missing endpoint, proxy, rate_limit_per_hour (contributed by helper.RestConnection on the live struct). - Service was missing scope_config_id (contributed by common.Scope). - ScopeConfig was missing connection_id and name (contributed by common.ScopeConfig, which the archived base type does not include). Fold the missing fields into the archived models so a single init migration produces the correct schema. Since rootly has not been deployed anywhere, keeping one init migration is cleaner than chaining follow-up ALTERs; a fresh migrate creates the correct tables in one pass. * fix(rootly): align API contract with real Rootly /v1 responses The original plugin shape was built from docs summaries rather than an actual response capture and diverged from the Rootly API in ways that silently produced zero incidents. Reconcile every code path with ground truth from the OpenAPI spec and a captured GET /v1/incidents response: - Connection test hits /users/me (Rootly's real "who am I" path); the original /users/current returns 404. - Incident list filter is filter[service_ids]=<uuid>, not filter[services]=<uuid>. The latter exists but accepts names and silently matches nothing for a UUID. - Role-bearing user fields (user, started_by, mitigated_by, resolved_by, closed_by) and severity are JSON:API response envelopes nested on attributes: {"data":{"id":...,"attributes":...}}. The previous flat NestedUser / SeverityAttrs shapes were reading the wrong paths, so those fields were always empty. - Service membership lives on the sibling relationships block as JSON:API id+type pointers, not on attributes. The safety-net scope-filter check now reads from the right place. - The incident resource does not have an urgency field. Drop the corresponding column from the model and archived schema. Also harden the collector: split the ResponseParser / next-page logic so pagination state is captured during parse (rather than re-reading the already-drained response body in GetNextPageCustomData), and add lightweight request/response diagnostics gated behind Debug logging. Verified end-to-end against a live Rootly tenant: 3 of 6 scoped services returned incidents, all 3 extracted and converted into ticket.Issue rows with creator assignees and board linkage. * style(rootly): trim narrative comments to match plugin conventions Match PagerDuty's comment density. Keep the few comments that flag non-obvious invariants (archived-base field overrides, 1-based pagination, deliberate divergence from PagerDuty's panic-on-unknown behavior, clock-skew guard). * refactor(rootly): address code review feedback Apply fixes from a multi-lens code review pass: - API: rename swagger {serviceId} to {scopeId} to match registered route; remove dead Proxy handler. - Models: add Sanitize() on RootlyConn; add RoleUserIds() helper on Incident; index ServiceId; drop unused Url field on User; remove dead RootlyResponse/ApiUserResponse types. - Migrations: mirror live schema in archived models (index on service_id; drop user.url). - Collector: switch pagination to reqData.Pager.Page (avoids divide-by-zero), cap at 10000 pages, extract buildIncidentsQuery as a pure helper, drop unreachable lastPageEmpty branch and unused TotalCount, remove diagnostic logs; add unit test pinning the filter[service_ids] param literal as a regression guard. - Services: preserve ScopeConfigId across re-collections; declare ProductTables on collector and extractor metas. - Extractor: skip emitting User rows with neither name nor email so sibling scope tasks can fill in fuller data; use generic resolve() for SequentialId; type ServiceRef as a named struct. - Converter: consolidate mapStatus to return (mapped, known); use Incident.RoleUserIds() instead of an inline slice. - impl.go: comment justifying services-before-incidents subtask ordering. - e2e: rewrite raw incident fixtures to JSON:API envelope shape; regenerate snapshots (drop urgency column). * feat(rootly): add rootly dashboard * chore: run gofmt * chore: add isBeta flag to rootly plugin * fix: Multiple credential-leak paths in gitextractor (#8872) * fix(asana): implement encryption for Asana connection token (#8873) * fix: harden dbt pipeline inputs (#8878) * fix: harden dbt pipeline inputs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(test): disable auth for local e2e server Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Klesh Wong <kleshwong@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(push-api): implement push API with authentication and validation logic (#8879) Co-authored-by: Klesh Wong <kleshwong@gmail.com> * fix: require secret for forwarded user auth (#8880) Reject spoofed X-Forwarded-User headers unless a configured shared secret matches the forwarded secret header. Co-authored-by: Klesh Wong <kleshwong@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: upgrade go to 1.26 (#8875) * chore: upgrade go to 1.26 * ci: update github action versions * refactor: replace deprecated go APIs --------- Co-authored-by: Klesh Wong <klesh@qq.com> * Allow Rootly plugin to collect incidents without service identifiers assigned (#8892) * fix: map Rootly 'completed' status to DONE in incident converter The mapStatus function was missing 'completed' as a terminal status, causing incidents with that status to be written as IN_PROGRESS instead of DONE. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: use updated_date as MTTR fallback for Rootly 'completed' incidents Rootly's 'completed' status represents a fully resolved incident that has gone through post-incident review. These incidents may not have a resolved_date populated. Fall back to updated_date so MTTR is computed correctly instead of showing the incident as unresolved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: remove service filter requirement from Rootly incident collection Allow collecting all incidents without filtering by…

dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. pr-type/bug-fix This PR fixes a bug severity/p1 This bug affects functionality or significantly affect ux labels Mar 19, 2026

fix(sonarqube): increase cq_issues and cq_file_metrics project_key le…

df0b5a9

…ngth to 500 Fixes apache#8331

jbsmith7741 force-pushed the sq_data_too_long branch from 692fbb5 to df0b5a9 Compare March 19, 2026 15:05

klesh approved these changes Mar 23, 2026

View reviewed changes

klesh merged commit c29ffc0 into apache:main Mar 23, 2026
10 checks passed

la-tamas pushed a commit to archfz/incubator-devlake that referenced this pull request Mar 26, 2026

fix(sonarqube): increase cq_issues and cq_file_metrics project_key le…

fd4d572

…ngth to 500 (apache#8783) Fixes apache#8331

la-tamas pushed a commit to archfz/incubator-devlake that referenced this pull request Apr 9, 2026

fix(sonarqube): increase cq_issues and cq_file_metrics project_key le…

af806ee

…ngth to 500 (apache#8783) Fixes apache#8331

jbsmith7741 deleted the sq_data_too_long branch June 22, 2026 17:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(sonarqube): increase cq_issues and cq_file_metrics project_key length to 500#8783

fix(sonarqube): increase cq_issues and cq_file_metrics project_key length to 500#8783
klesh merged 1 commit into
apache:mainfrom
jbsmith7741:sq_data_too_long

jbsmith7741 commented Mar 19, 2026

Uh oh!

klesh left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jbsmith7741 commented Mar 19, 2026

Summary

Does this close any open issues?

Screenshots

Other Information

Uh oh!

klesh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants